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Titlf^f the. Invention 
File System Image Transfer 
Background ^ Invention 

Field of the Invention 

The invention relates to storage systems. 



2. Related Art 

,„ computer file systems for storing and retrieving information, it is 
sometimes advantageous ,„ duplicate a„ or part of the file system. For — one 
purpose for duplicating a file system is to maintain a backup copy of the file system o 

lec, agahtst lost information. Another purpose for duplicating a file system - 
provide replicas of the data in mat file system available a, multtple servers, to be abie 

share load incurred in accessing that data. 

One problem in the known art is that known techniques for duplicating data 
in . f„e system either are relatively awkward and slow (such as duplication to tape), or 
1 relaUvely expensive (such as duplicafion to an additional set of dtsk dnvesX F 
example known .echoes for duplication to tape rely on logical operates of the file 

discourages frequent use, resultmg in backup copies tha, are relafively sta 1. ~ 
„ iost, the most recent backup copy might then be a day old, or several days old, severely 
reducing the value of the backup copy. 

Similarly, known techniques for duplication to an additiona, se, of disk 
drives rely on the physical forma, of the fi.e system as stored on the original set of .sk 
n Ueseknowntechniques use an additiona, se, of disk drives for duphcattonof* 
file system. Being relative* expensive discourages use, particular, for large file 
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systems. Also, relying „„ ^ ph jca| 

known ^ f ; r rr mp ; ,,y ° r ^ •** - -»* * * 

— JT^ZT. ' : h,c " consisttm copies of * — - 
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^agfDgscri^on of the Drawing 

^ ' Sh0WS 3 ^ *— 1 of ' ** - «- sys tem image 



- — imag :rL 2 shows a b,ock d ^ ° f a - ° f « - ■ — - * 
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transfer. 



Figure 3 shows a process flow diagram of a method for file system image 
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,„ the following ascription, a preferred embodiment of the invention is 
scribed with regard to preferred process steps and data structures. However, those 
slti „ed in the art would recognize, after pen.sa. of this application, tha, embodtments of 
lhe invention may be implemented using one or more general purpose processors (or 
sp ecia. purpose processors adapted to the particu.ar process steps and data structures) 
operating under prog™ control, and that implementation of the preferred process steps 
1 data structures described herein using such eouipmen, would no. reoutre undue 
experimentation or further invention. 

mentions described herein can be used in conjunction with inventions 
described in the following applications: 

„ Application Sena. No. OS/471,218, fled June 5, 1995, in the name of inventors 
David Hi* e. al„ titled "A Method for Providing Parity in a Ratd Sub-System 
Using Non-Volatile Memory", attorney docket number NET-004; 

„ Application Serial No. 08,454,921, fled May 31, 1995, in the name of inventors 
D l Hi. e, a.., titled "Write Anywhere F«e-Sys.em Layout", attorney docket 
number NET-005; 

!5 0 Application Serial No. OS/464,59!, filed May 31, .995, in the name of inventors 
Zi Hit* e, a.„ titled "Method for Allocating Files in a File System Integrated 
with a Raid Disk Sub-System", attorney docket number NET-006. 

Each of these applications is hereby incorporated by reference as if fully se, 
' forihherein. They are collectively referred to as the "WAFL Disclosures." 
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image transfer. 
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FikS °™°™lF ikS y stem , mageTraKfet 

5 A system 1 00 for file svstrm ;™ 

destination ^ 12 „ WB " «— *r inc.udes a fi,e server , ,0 and . 

The file server 1 10 inc , udes 
■"emory 1,2, a„ d mass s| P ^ • *t of program and data 

bribed » the WAFL Disclosures' T„ ^ ^ 3 ^ U ° ^ « 
includes a RAID storage subsystem. ' " " emb ° diment - «* — borage 1,3 

The destination file svstem on • , . 
— * a magnetic or optica, d J * * " J * - ~» - as a fl as„ 

5 - destination i. "^T^ * 

-bsvsten, Thedesdnationfflesvs^,,. f ° mC ' UdCS 2 -«»•. 

— ,,o usingacommuni.,::;;:;: coup,ed direc,,y ° r — * <° - 

•0 » me s ^7IZ^Z7' deS,i " a,i0 " fi ' e ^ " ~>~ 

>» for the flle server , ,0, sueh as an "**> " **- 

as an I/O bus, a me^ine bus , or other ^ ^ 

In a second preferred ^mu^ 
-.udedinasecondmeselr,, T "* «» * 

«"» > >0, incudes a processor , 1 , Z!T ^ ' '°' *** "* ta fl " 
— "3 ma, serves as me Z T"* ^ " ^ 

server 1,0. Tie second file server „, ^ /T 10 *« fol * 

bribed i„ me WAF L Di ^'^ ^ '>« lik e „ ne 

^^''""^ISOmoludesanetworkLlr 0 "?^ emb0<li,,,em ' 
second fi,e server 1,0 suchasad' * ^ " etWeen lhc ^server , 10 and me 

a WAN (wide area network) a NlZ «* a LAN (he ^ nelworl()i 

■*»*), a NUMA network, or another interconnect 
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ln a ted preferred embodimen, .he communication path 130 inc.udes an 
intermediate storage medium, such as a rape, and ,he destination fi.e system .20 can be 
ZTL me -er , ,0 Wf or a second f,.e server U0. As shown be.ow, when 

?20 let of storage blocks can be transferred by storing .hem onto the inte— 
5 120, that set oi sior g from ^ 

storage medium. At a later time, retrieving that set of storag 
intermediate storage medium completes the transfer. 

It is an aspect of the invention that there are no particular restrictions on the 
• a i ™ For example a first part of the communication path 130 can 

• — r: :r zz — - - . — — 

path 130 can include an intermediate storage medium. 

B is a farmer aspect of the invention that the destination file system 120 
,aa- .he first file server 110, in a second file server 110, or disu.bu.ed 
15 '""I To " 0. TransferofstoragebloCfiomthefirstfileserver 
r:t— ! Isystem ,20 ,s *us completely genera,, and includes the 
possibility of a wide variety of different file system operate 

„ Storage blocks from the firs, file server 110 can be dumped .o an intermedia* 
20 ^medium, such as a. ape or a second disk drive, retained for a penod of 

;;: 8 and — » * «- - — 1 * ^ - - f,,c ^ 1 10 

can itself be the destination file system. 

, o Storage blocks from the firs, file server 1.0 can be transferred to a second file 
M re!.,0,andusedatma,secondfi,ese„erUO.T»us,mes,orageb„cscan 

Storaae Mocks from the first file server 1 10 can be distributed using a plurahty of 
o Storage blocks tr ,..,„„ m some of the storage blocks are 

different communication paths U0, so that some 
30 oinerci" ^ . n»lativelv slow intermediate 

immediately accessible while others are recorded in a relatively 

storage medium, such as tape. 
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° to ta flie server ll0canbe , 

Wm, transferred using ^ .'° *" S " m ^ • complete file 

8 me communication path i 30 anrf tk. 
a complete file sysKm a , thed uu - md *=n processed to form 

me destination file system 120. 

In alternative embodiments descrih»H i. • 
<-e a second destination file system J " ^ "* — 4 «" T HO can 

be included within me second fiie serve, . 10 '* C '"" 1 * SyStem ,20 <*" 

"0 similar ,„ , he fi ret file ^ , , "" h ilK,u ««< a third file server 

«er noor the second file server 110. 

More generally, each „» fife server l in . 
'20, either included within the „* file server nn * * sttm 

"0. The se, of fife ^ , „ " '°; ° r «*— an „+,» a, serV£r 

«« server 1 ,0 as the root of mat ° 2 ~* a tree with the 

15 File S y ste "> Storage Blocks 

As described in the WAFT tv t 
"» (- in genera,, on the „» fi, e ^ 2 *> »« « the fi,e seiver 

of which is stored either in the memory U2 m „Tl * " " *"* ** ' ' * "* 
» I M includes a current hick map whil r . 7 1 a ^ fite 

As described in the WAFF n««i 
"3 is a, a,, times consent T„ U s ' * ° n "» — '"rage 

As used herein, the term "consistent » „, ■ 
^rage Nocks to . file ^ * a file system („ r ,„ 

-tades a„ b,„cks reouired for * data J n ** SySKm te 

» «— » file system stands on its £ ! " ""*" ° f ^ * ™* a 

systemat ^-----o. h ^::re:i:rr a state of ae * 
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As described in the WAFL Disclosures, when changes to the fi.e system 
„4 are commuted to the mass storage .13. .he block map is altered ,o show , hose 
storage Hocks U5 that are par. of the commined fi.e system 114. In a preferred 
embodiment, me f..e server U0 update, me file sys.em freouenfiv, such as about once 



5 



each 10 seconds. 
Snapshots 



Figure 2 shows a block diagram of a set of snapshots in a system for file 

i o system image transfer. 

As used herein, a "snapshot- is a set of storage blocks, the member storage 
blocks forming a co.is.en. file system, disposed using a data structure that allows or 
se, management. The efficient set management can include time efficency or 
„ ^operations (such as logical sum, logical difference, membership, add member, remove 
' 5 I 1, F or example, the time efficiency can include 0,n) time or less for n storage 
Mocks The efficient se. ma.gemen. can also include space efficiency for enumerating 
ta set (such as association with physica, location on mass storage or inverttng *e 
me mbe,ship funcion). The space efficiency can mean about 4 bytes or less P r 4K 
l g e block of disk space, a ratio about ,000:1 better than dupUcatmg me storage 



20 

space. 



As described herein, the data structure for the snapshot is stored in the file 
sys ,em so mere is no need to .averse the file system tree to recover it. In a preferred 
25 embodiment, each snapshot is stored as a fi.e system object, such as a blockmap. Th 

to identify if the storage block is in the active fi.e system. 

Moreover, when the file system is backed-up, restored, or otherwise copied 
30 or transferred, One blockmap within the fi.e system is as par, of the same operation .tse 
also backed-up, restore* or otherwise copied or .ransferred. Thus, operattons on the fi.e 

system inherently include preserving snapshots. 

7 
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Any particular snapshot can be 

technique, including 6 transferred any communication 

° "** Stor ^ ^ an intermediate storage me „- , 

—ory, tape, disk in th e same fifc system < SUch - nonvolatile 

distributed over several file system ) ' " 3 «° « ** 



transfer using one or more network 



messages, 



10 o transfer using communication within a, i™. r. 

- d isk dislnbuted over several filc ^ * ** - . flle systera , or 

colun» indicates . pUe «-» bb*. , ,5. !„ fte figure , each 

borage Mock „, (thal , he - — row i»dica,es 0IK 

, . , tory of ""I storage block mi.- • 

excluded from successive snaps „ 01s . ge b,Mk »«ng mcludcd in or 

- .here is a bi« 2 „ indicating whefcer t^ ' "°" * "* COiUmn ^ ~" 

■ha. particular snapshot 210. ^rage block „ s is inc|uded h 

- . fllc ;::ri;:::rp:r; r — s,o - e - 115 

— POin. in time. A s^no, 210 ^ be J** ^ ( «— « * . 14 « 

•™e by copying the bits from the block man • 7 " ' a, " ,, ~ '° map al ■» 
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• tpnt file system 1 14 that is being used and perhaps 
modified, while the snapshots 210 represent F 



read-only. 



. ^ file - — ~ * SnaPSh0B t 21 ° ^ 

0 ^re is only one active file system .* — - - - ^ * 
multiple snapshots 210. 

, ■ ,h, file server 1 10 creates a new bit plane, in response to 
« selected tunes, the file .rv r • 

mc bloc, map, to create a new snaps * 2* * „ £w 

^ t „r backup and mtrronng of *e «n ^ ^ „ 
snapshots 210 are created at penodtc tunes, such as on P 
5 as otherwise directed by an operator of the file server 1 10. 

Storage Images and Image Streams 

- M -^ in T e :::::-^oTsipsh:, ^< 

• „ A .et of associated block locations for those storage 
2S from a storage ,m,g. A s« » * ^ ^ ^ or implicW 

th e storage .mag can b « ^ „ be idcntifl ed explicitly by 

For a first example, the set of associate ^ 

^^™^^*^*Z£* ZL in which the 
of associated bloc, locations can be ident.fi* ^ - withto te image 

J0 ^ragc blocks from the storage image arc pos.uoned or transferr 



stream. 



9 



WO 00/07104 

PC17US99/17148 

In a preferred embodiment th 
- "W* of an image stream ^ " is op,i mi2ed so 

-to is „p,i rai2ed . ,„ f «-» '"-age s.ream from one file sma to 

~* blocks idenMed . n fc J™* »f storage blocb . sdec|ed ^ ^ 

P-W from a p Iura , ily of didB ta « * « * - much . ^ copjed ^ 

10 ^ the tet fUe ^ s ^ - as ,„ raaximi2e ae 

A storage image 220 c 

^e fi ,e System 1I410thedes ;z°[;r s ° Wocks 1,5 ,o 66 

The storage blocks 115 in the • 
<-=y can be combine, , 0 J 220 are selected so that whe „ 

- ^ combined wilh storage storage ira a ge 220 tfc ,, „ 

•referred a, earlier toes). ° m 0ther ""W ^ges 220 (which were 

As shown herein, the file server nn 
response to one or more snapshot 210 ^ s,ora * e ""age 220 in 

A" image stream 230 comonV, = 
s«°-ge image 220. When the ^ J ° f ■** „5 from . 

-rage h,„ cks „ , „ ^ ^ ^ • »> - copied from the file system 1 H, tt e 
information. When the image ^ 2J « "° with block locatio „ 

* storage blocks 1 ,5 ia * ilMge « «* ««,ir.a,i„„ ak system ^ 

,20i "~,o,he b , MklocaUon . fo ^ - »« ^,i„a,i„„ flle 
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Image Addition and Subtraction 

The system 100 manipulates .he bits 21 lin a selected set of storage images 
220 to select sets of storage blocks 1 15, and thus form a new storage image 220. 

For example, to following different types of manipulation are possible: 

, The system 100 can form a logical sum of two storage images 220 A + B by 
, forming a set of bits 2,1 each of which is to logical OR (A v B, „ to 

corresponding bits 2,1 in the two storage images 220. The logic, sum of two 
storage images 220 A + B is the union of those two storage .mages 220. 

0 The system ,00 can form a logical difference of two s.orage images 220 A - B by 
forming a se. of bits 21, each of which is logic. - V only ifthe coring >. 
2 7a is logical "1" and .he corresponding bit 2.1 B is logical "0" .„ the two 

storage images 220. 

The logical sum of mo storage images 220 A + B comprises a storage 
20 taage 220 that includes storage blocks ,15 in either of the mo origina, storage images 
I Using the iogica, sum, to system .00 can determine no, Jus, a single pas, state of 
to fllTsyLm 1,4, bu, a,s„ a hisiory of pas, states of tot flle system ,M « were 
recorded as snapshots 210. 

The logical difference of mo selected storage images 220 A - B comprises 
M just those storage blocks that are included in to storage image 220 A but no, in to 
storage image 220 B. (To preserve integru, of in— 1 storage .mages, to 
Zhendsorageimage220Bisalwaysasnapsho.2,0, A logical difference rsusef. 

su „™ u • o c^t nf storage blocks forming an 

for determining a storage image 220 having a set of storage 

30 incremental image, which can be used in combination with full images. 
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- - ~ -r;:^:: * - 7- - raay 

7** 2, °' S ° - '° w„ic„ SIOrage J"™"' 3 *** of 

those snapshots 210. 115 were not changed between 

5 

In further alternative embod* 
211 from each snapshot 210 f 0r other ou ^ ^ ^ *° USe the bi " 

the storage blocks „ 5 represented by Jj^^ * t0 operations on 

10 Incremen M Storage Images 

As used herein an "in 
-ee nafelaorageimageMd J~ r=i ,a g , „ . logical dlff _ 

called the «t op « storage ^^^^"^ * ~ *' St0rage ima § e 2 20 A is 
storage image 220. ' image 220 B » called the "base" 

When the base storage imaee 220 r 
20 "5 in a consistent file system 114 °™ aM ~ F ^^Mo* 

incremental changes to the file system i,4b A ~ B inC,Udes those 

top storage image 220 A. ""^ ^ baSe St ° rage ima S e 22 <> B and the 

Each incremental storage image 2^0 . 

25 aorase 22 »- 1«— « s torae e „* 20 ; '° p s 7 8e image 220 - * >- 

» a sequence of aorag e images 220 C h Whe " *« 

•op storage image 220 for a next O ' " S '° ra8e ""^ 220 for 

each C f is a 



/ / / 
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Examples of Incremental Images 

For a flrs. example, .he system 100 can make a snapshot 210 each day, and 
form a level-0 storage image 220 in response to the logical su m of daily snapshots 210. 

June3.1evel0 =•= June3 + June2 + Junel 

(JuneS, June2, and Junel are snapshots 220 taken on those respective dates.) 

The Junc3.1evel0 storage image 220 includes all storage blocks 1 15 in the 
daily snapshot, 2.0 June3, June2, and June.. Accordingly, the June3..cve,0 storag^ 
imag e 220 incudes all storage Hocks 1.5 in a consistent file system 1.4 (as well - 
possibly other storage blocks .15 mat are unnecessary for the consistent file system 1 
active at the time of the June3 snapshot 210). 

In the firs, examp.e, me system 100 can form an (incremental) level-1 

difference with a single snapshot 210. 

JuneS.levell = JuneS + June4 - June3 

(Junes, June4 and June3 are snapshots 220 taken on those respective dates.) 

ft is no. required to subtract me June2 and Junel snapshots 210 when 

forming ,he JuneS.level, storage image 220. All storage blocks 1,5 that the JuneS 
s „ aps ho,2,0a„dthe J unc4snapshot2 1 0haveinc„mmonwimei,hermeJu^ 

25 2,0 or the June, snapshot 2,0, toy will necessarily have in common w..h the Jun 3 
snapshots. This is because arty storageblock 1.5 mat was par. of me file system 1.4 
rie2 or June,, and is stUi par, of the f,e system 1,4 on JuneS or Ju„e4, must have 
also been part of the file system 1 14 on June3. 

to the M example, the system ,00 can form an (incremental) leve.-2 
st „ragc image 220 in response to the logical sum of daily snapshots 210 and the lo g .ca, 
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difference with a single snaoshot J 1 n f 

apsho, 2,0 from the lime of (he , evel , ^ 



h the first example, the slorag e ; , 

^..eveO e„„ec ti ve, y i„ c , ud e al , s,„ J b: ^ 3 " Jl " ,C5 •' eVe "• ^ 

borage Mocks ■ ,5 in , consisttnl fi , e ^ * ' 5 ~" » ~, a ft,, „ F of 

For a second example, the system inn M e 
storage image 22 „ b respoilse , o 7 - *« • **« On™, 

'ogica, difference with . single IJ f " ' » ° f 2,0 a nd the 

220. P ' l0frOm,he " me ° f ^'eve|.0st„ ra ge image 

15 

,une9levell =June9 + Junes -j u „e 3 

» — sr: srjirjrr 220 jm ~ - 

biocKs „5 in a ^ fifc J °< * »' *> ~ a full set F of 
that the June9.,evell storage imase «n h , ' ' ' S n ° particular ^rnent 
^.^storageimageLin:;!:, ^ * ** ** 

File System Image Transfer Techniques 

To perform one of these mnvi.. 

* transfer** data ftonl _ ^^^^^ 
destination file syst em 120 H „ I ""mnunication path 130 to the 

- » - i- .h?i::;::r:r - - - - ^ 

y «m 114 to the corresponding destination file 
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system 
server 110 



120 without logical file system processing by the file system 1 14 on the first file 



5 



10 



15 



20 



In a preferred embodiment, the system 100 is disposed to perform one of at 
least four such copying operations: 

o Volume Copying. The system 100 can be disposed to create an image stream 230 
for copying the file system 1 14 to the destination file system 120. 

The image stream 230 comprises a sequence of storage blocks 115 from a 
storage image 220. As in nearly all the image transfer techniques described herein, that 
storage image 220 can represent a full image or an incremental image: 

Full image: The storage blocks 115 and the storage image 220 represent a 
complete and consistent file system 1 14. 

Incremental image: The storage blocks 115 and the storage image 220 represent 
an incremental set of changes to a consistent file system 114, which when 
combined with that file system 114 form a new consistent file system 1 14. 

The image stream 230 can be copied from the file server 110 to the 
destination file system 120 using any communication technique. This could include a 
direct communication link, a LAN (local area network), a WAN (wide area network), 
transfer via tape, or a combination thereof. When the image stream 230 is transferred 
using a network, the storage blocks 115 are encapsulated in messages using a network 
communication protocol known to the file server 110 and to the destination file system 
120 In some network communication protocols, there can be additional messages 
between the file server 1 10 and to the destination file system 120 to ensure the receipt of 
a complete and correct copy of the image stream 230. 
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The destination file system 120 receives • 
identifies the storage blocks 115 f r „ * 230 md 

5 When the storage blocks 115 reoresent » * , 

„4. me destination file system 120 « 7 " f " e 
chanse. tte desti.tion fl ,e sys, m ,7o 71 ^ ' " ^ **« 

on,y access „y ,ocal p r o=esse I , ' ' " ^ * 

- ,,, l :ir a : d : mati °" * - 

io making changes by those lnr»i * Pr ° CeSSeS ' without 

ges by those local processes available to the file server 1 1 n th 
source of the file system 114. 110 that was 1,16 

When the storage blocks 1 1 s r^. 
» <* «.e system ,H form . „ ew consistem f „ e ^ ^ 

include fl ,elT«: f IT ' 14 ^ " h - *» "0, 

^ 1 ~r y - ^ me — * ~ « 

- f-e system 1,4 " ^ ^ 210 — » 

- "disk co^^ T ^ ™ b ° di - ° f «*» «W* (ne rei „ 

^z^rz:rz m - r a disk - ^ 
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When performing disk copying, the system 100 creates an image stream 
230, and copies the selected storage blocks 115 from the mass storage 113 at the file 
server 110 to corresponding locations on the destination file system 120. Because the 
mass storage 1 13 at the file server 1 10 and the destination file system 120 are both disk 
5 drives, copying to corresponding locations should be simple and effective. 

It is possible that locations of storage blocks 1 15 at the mass storage 1 13 at 
the file server 1 10 and at the destination file system 120 do not readily coincide, such as 
if the mass storage 113 and the destination file system 120 have different sizes or 
,o formatting. In those cases, the destination file system 120 can reorder the storage blocks 
115 in the image stream 230, similar to the "Tape Backup" embodiment described 
herein. 

Ta pe Backup. In a second preferred embodiment of volume copying 
15 (herein called "tape backup"), the destination file system 120 can include a tape device or 
other similar long-term storage device. The system 100 can copy storage blocks 115 
from the mass storage 113 to that long-term storage device, providing a backup copy of 
the file system 1 14 that can be restored at a later time. 

When performing tape backup, the system 100 creates an image stream 
230, and copies the selected storage blocks 115 from the mass storage 113 at the file 
server 110 to a sequence of new locations on the destination file system 120. Because 
the destination file system 120 includes one or more tape drives, the system 100 creates 
and transmits a table indicating which locations on the mass storage 113 correspond to 
25 which other locations on the destination file system 120. 

Similar to transfer of an image stream 230 using a network communication 
protocol, the destination file system 120 can add additional information to the image 
stream 230 for recording on tape. This additional information can include tape headers 
30 and tape gaps, blocking or clustering of storage blocks 115 for recording on tape, and 
reformatting of storage blocks 1 1 5 for recording on tape. 



20 
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- - r ment of voiume — 

««- 1I4 , either at the file :r ;: f e copied to a new fi,e — ■ * 

system 120. ^ 1 10 or at a file system 1 14 on the destination file 

Similar to tape backup, the destination file svstem 1 90 „ 
Nation to the ima g est r ea m23 o forrecording ^ 

can include file metadata usefil l for the file sysZZT 

within the file. * 1 14 t0 ,0cate stora ge blocks 1 15 



10 



0 ^ynej^irrc^ , J . 

second le Z Z T '° deS,ina,i0 " fi ' e 120 '° » 

system , ,1 " 3 feqUCm b2SiS ' *» « • co Py of the f„e 

-•-^i^^: e r fc T rf,, " ,,b '-- ,,4 - b 

example S ,*e Hrs, file server , 10 ^ ' '° *" "* fa «« > ">■ «* - for 

When performing volume mirroring the svstem inn r 
i ^es.eam230repre S e„U„ gacomptaefi|esyst ^ 4 7 e «. r, ^ " " 
destination file system ,20. Th. sysL 10 „ I , 
230 represent „,„ J^ZT " ^ *** 

*e destination file system ,20 T d """" ^ * "° * 

- * seance of increment, i raage ~ 'i*" * " — — » 

-age image";;: 0 ::;':' 0 perform vo,ame mimrins using - « 

mage 230 and a sequence of incremental storage images 230 H„ 
toe ™» in ^e storage biocks! 15,„ be included in an' ! ' 

can take substantia, time for a re.ative,' It ' ^ 
subtraction " 86 1 14 ' if *« "X logical 
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As used herein, a «mark-on-al>ocate storage image" is a subset of a 

originally formed a consistent file system. 

to a preferred embodiment, rather than using logical subtraction, as 
ascribed above, a, the time the in— storage images 230 is about to be 
Iferred, the file server 1,0 — a separate "mar—t, storage ^ 
230 The marlc-on-allocate storage image 230 is constntcted by settmg a b,t fo each 
„ock„5 as it is added to the consistent file system .14. The mark-on-allocate 

zz - - - - - * — - *■ - ^ nx inciuded in *: 

rfniap, or otherwise backed-up; i, can be reconstructed from other storage tmages 
230 already at the file server 110. 

oift u transferred, a first mark-on- 
When an incremental storage image 230 is translerrea, 

• oe 230 is used to determine which storage blocks 1 15 to include m 
; allocate storage image 230 is usea to uci 0 „„?™isused 
• 7^0 for transfer A second mark-on-allocate storage image 230 is used 
the storage image 230 for transier. a 

t0 record changes to the file system ,H while the transfer .s **»£^L* 
^sfer is performed, the first and second mark-on-allocate storage tmages 230 g 



roles. 
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25 



t^^. ta a first preferred embodiment of volume mirroring 
(hK ein caned »f»„ mirroring"), the destination fde system ,20 includes a d,sk dnve 
other similar accessible storage device. 

Upon the initial transfer of the full storage image 230 from the file server 
U0 , the destination fi,e system ,20 creates a copy of the — » - 
Unon Ute sequential transfer of each incremental storage .mage 230 from the 

file .stem ,20 updates , copy 
The destination file system 120 thus maintains its copy of the file syste 
to date, and can be inspected at any time. 
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When performing full mirrorina 
creates an image stream 230, and copies th , C ° Pying ' the System «» 

113 at the file server , " ^ -rage blocks 115 from ^ 
^stem ,20. corresponding locations on the desnn ation file 

5 

^^^fflentaMvlir^^ In 
"Coring (herei „ called ., jncrementai J^**™* of volume 

^oothO, a tape device or oth er relatively [ ' "* *"""*» fik 120 can 

«her re,a,ive,y fas, storage ^ *" and (2, . disk drive 

10 

As used herein an «;~~ 
-ge image fr „m,he tafil ^rrr;^ " ' *« * — * a base 
- «.e system, „„ l~ a ' — ~, storagetaagefrom 

con,p,e t ec„p yofthefa(ffle J J* 3 ° f ~al.y differ, .ype, ^ , 

can be rec„„ slructed from ^ ^ ^ ^ ^ 

Upon the initial transfer of the fi,n 
' the destination f„e system 120 copjes , *" "»* ™* *0 fr„ m ^ ffle ^ 

— *»* 1 -3 to that relatively JZ ? * ** ° f b '°* " J the 

, . y siow storage devirp n~ *i 

«ch mcremental storage image 230 from * n SeqUeMii " '"«*' of 

° — - „ f storage ;r liir r des,ina,,on fi,e *- 

^..veiy fas, slorage e ^ the mass slorage ,„ ^ 

incremental sets of storage blocks U5 n • S '° ra8e ^ " 5 " lus 

' « - * no, retire an entire dup„ca,e " " " 

•he system mZ^^ZZZ T* " "* ta * 23 °. 

from me maS s st0 ra g e , I3 al me flIe ^ -orage blocks U5 

— storage device. The system ,00 writes l'" " ~" l0Ca "' 0 " S °" 

<** location inform^, t0 ^ ^ including s torag e 

en.bodime„,, m e system 100 uses a ^ * W ™ ^ *> * Purred 

» "hat the base storage image 230 J* " T^""" dK,i " a,i0n "«* 
having ,„ occupy disk space . *" " Substoti a' Period of t i me withou| 
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p„ r each incremental storage image 230, the system ,00 creates a new 
im age stream 230, and copies the se.ected storage blocks , 1 5 from the mass storage 
Te l server 110 to a set of new locations on the accessible storage device. 
Lie!:, Ige i^es 230 are created con— and —y a, P er,od,c 
times that are relatively close together. 

incremental storage images 230 are received at the destination fie 
^m ,20, which unpacks them and records the copied storage 

copied storage blocks 115 overwrite the equivalent storage blocks 115 rom ea 

, imaees 230 In a preferred embodiment, the incremental mirror data 

incremental storage images liv. in a p. . , w , , « ,>,,, are 

different from the base storage image 230. 

In a preferred embodiment, the incremental storage images 230 are 
emitted to the destination file system ,20 wuh a data satire . set o 

Hocks 1,5 that were deallocated (that is, removed) from the file system 

a t « blocks 115 from its incremental mirror data structure. This 
the f„e server 1 10 and me base storage image 230 from the file server HO. 

c^raaP*. ">» cither m mirro ' ing " inCrementa ' 

* c <u r,\o carver 110 to update its consistent tile system 

:„:::;:.i i — — , - *— - 

further detail in the WAFL Disclosures. 

J u A- m * n t the file server 1 10 does not attempt to create a 
In a preferred embodiment, the tile server 

• aee 230 and to transfer storage blocks 115 for every consistency pomt. 
storage image 230 and ^ ^ mQSt 

Instead, after a transfer of a storage image 230, the file serve 
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<*- accessing storage device. ^ ^ ' 2 ° « a disk drive or 

"» s,o rag e , ,3 «„ accessi . . J 3 '™ 100 CM C °^"»*ge Weeks f rom fte 

The file server 110 maintain 
master snapsho, 210 is a snapshot 210 J™* ' ° f Se ' CC ' ed masto ™P=*«B 210. A 

- — .20, so * a , fte 1J * * des„ na ,io n 

- fl« «-» H4 maintained a , (hc ^ ™ 20 - - whh reference 10 

» maSttr ^o.2 1 0 isd esig„ atedbyanoDeratn °' 3 P refe -d embodtaem, each 

In a preferred embodiment at a „■ • 
-ined u„,i, all dest] . nat . ^ » each n^r snapsho , m fa 

» -Psho, 210. A master sn3psho( 2 , „ J"™*. ' 2 ° ^ pas, thal master 

shadow snapsho, 2,0. Tha, is destinati ,, ™ 8 0f T 

^ o f Aa , mas , er shadow s ; ap - * ^ , 2 „ wait for completion of ^ 

*» <na<n,as,ershadows nap " " a "° Wed '° ^ « 

25 

destination file system 120 generates , 
command ofan operator or in responses - v v """^ (SUCh as u P on 

newest master snapshot 210 10 which ,1, „ • 7 ' K ™ e messa Se includes a 
- synchronic nt hC «<e «~a .20 has most r ec ently 

snapshot 2,0. ""° """^ *« - no such newest master 
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The file server 1 10 determines any incremental changes that have occurred 
,„ the file system 1 14 from the newest master snapshot 210 a. the destination file system 
,20 to the newest master snapshot 210 a. the file server .10. In response to th,s 
determination, the file server 1 10 determines a storage image 230 including storage 
blocks 115 for transfer to the destination file system 120, so as to update the copy of the 
file system 1 14 at the destination file system 120. 

If there is no such newest master snapshot 210, the system 100 performs 
volume copying for a fitl. copy of the file system 114 represent hy the newest master 
snapshot 210 a. the file server 110. Similarly, if the oldest master snapshot 2.0 at the file 
server .10 is newer than the newest master snapshot 210 at the destination file system 
,20, the system .00 performs volume copying for a full copy of the file system 1.4. 

After volume replication, the destination file system 120 updates its most 
recent master snapshot 210 to be the most recent master snapshot 2.0 from the file server 
110. 

Volume replication is well suited to uploading upgrades to a publicly 
accessible database, document, or web site. Those destination file systems ,20, such as 

, mirror sites, can then obtain the uploaded upgrades periodically, when they are 
initialized, o, upon operator command at the destination file system 120. If the 
destination file systems 120 are no, in communication with the file server ,10 for a 
substantial period of toe, when communication is re-established, the destinauon file 
systems 120 can perform volume replication with the file server 110 to obtam a 

.5 substantially up-to-date copy of the file system 1 14. 

In a first preferred embodiment of volume replication (herein called 
"simple replication"), the destination file system .20 communicates directly (using a 
direct communication link, a LAN, a WAN, or a combination thereof, with the fie setter 



110. 
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*ec, co mmunication link , . LAN _ a ~^o mmumcates directly (us ,, g a 

destination fl,e sys ,e m ,20 Tfc second „ ', b ' na "° n """^ wi * a *«>„d 

nd dest »nation file system nn r. , 
> s ^r 110 to perform simp]e replication fnrfh r , System 120 **s like the file 

P replication for the fi rst destination file system 120. 

A sequence of such destination file systems 120 t , 

destination file system 120 ,h»» • dy teiminates iri a 

system 120 that communicates directly with th* n 

performs simple replication Th. lh Server 1 10 

P replication . The sequence of destination file system, f 
o replication hierarchy such a, in » ,r ♦ , y 20 thus forms a 

y. such as ,n a directed graph or a tree of file severs 1 10. 

In alternatiye embodiments, the system 100 can also perform 
combinations of these techniques. P 



one or more 



- ta p ;:r;: ;;: r snapsho,s 2)0 md havins - 

Reives. F„ r IT " ' «° - «- «**«. 210 

all destination file systems 120 Th „ „ ^ Simu,tane °usly for 

systems 120. Thus, all destination file systems im, 9 
the same master snapshot 210. synchronize to 



the same master snapshot 2 1 0. 
Shadow Snapshots 



storage blocks of the snaoshot din f ' ° ne ! ' me member 
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A shadow snapshot 210 has the property that the file server 1 10 can reuse 
th e storage blocks 1.5 in the snapshot 2,0 whenever needed. A shadow snapshot 2.0 
can be used as the base of an incremental storage image 230. In such cases, storage 
Mocks 115 might have been removed from the shadow snapshot 210 due to reuse by the 
file system 1.0. .. thus might occur that the incremental storage image 230 resulung 
from logically subtract using the shadow snapshot 2,0 includes b 
« are not strictly necessary (having been removed from the shadow snapshot 2,0 they 
are no. subtracted out). However, al, storage Hocks ,15 necessary for the incremental 
storage image 230 will still be included. 

For regular snapshots 210, the file server 110 does not reuse the storage 
Mocks ,15 in the snapshot 210 unti, the snapshot 2,0 is released. Even if the storage 
Mocks 1,5 in the snapshot 2,0 are no ,onger part of the active file system, the file server 

consistent file system 1 14 that can be accessed at a later tune. 

However, for shadow snapshots 2.0, the file server 110 can reuse the 
storag e blocks 1 .5 in ute shadow snapshot 2,0. When one of those storage Hocks 1 15 ts 

0 U5 Thus, each shadow snapshot 2,0 represents a se, of storage Hocks 

consistent file system 1 14 tha, have no, been changed in the active file system U stnee 
the shadow snapshot 2,0 was made. Because storage Hocks „5 
shadow snapshot 2,0 does not retain the property of represent a consM n« f,l « 
114 However, because the file server 1.0 can reuse those storage Hocks 1, the 
25 shad ow snapshot 2,0 does no. cause any storage Uodo 115 on the mass storage 
be permanently occupied. 

Method of Operation 

Figure 3 shows a process flow diagram of a method for file system image 

transfer. 
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A method 300 is nprform«,i • 

*— * *- <* : t: by n,e — »• - - 

described herein. fl ° W points ™* Process steps as 

5 / / / 

Generality of Operational Technique 

P^ftJo^T ^ SyStCm imagC tranSfCr the method 300 

10 

sys.e m , „ r copying or transfer ' " back "P - -core within the fe, flIe 
n s or transfer to a second file system. 

o ^^^^^^^ 

In the first (selection) steo th*» c 
-Piete fi,e system wclnbea ^ ^ °»» ™ selected can be a 

con.p.e,e fi,e system, st,ch M those ^ ' "*"« « te « "-men, , 0 ,„ £ 

an«her type of subse , ^ 8 hd » «« »°« been changed, or can be 

^kup operation, or repeated,, such f f 3 ** " f °' 3 

» - * selected in respond .^1^^ * 

P SS 3t 3 Send,n 8 «k ^rver or at a receiving file 

For example, as shown herein hW„ 

«* ^„p or copying o f an entire T ^ ^ ^ ^ «* 

system, or can for incrememal ^ ( 



a 

or 
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incremental coring of a f,.e system. The storage image 220 selected can be 
define, by a sending fi.e serve,, or can be defined in response to a revest by a 
receiving file server (or set of receiving file servers). 

,„ the second (operational) step, the image stream 230 can be selected so as 

^sfer ,„ different types of media, to optimize transfer rate, or to opfmrze rehabu ty. » 
, preferred embodiment, the image stream 230 is optimized to maximtze transfer rate 
from parallel disks in a RAID disk system. 

,„ the third (reconstruction) step, the image stream 230 can be 
reconstructed into a complete file system, or can be reconstructed into an increment of a 
I ystem. The reconstruction step can he performed immediate, or after a delay, can 
be performed in response to the process ma, initiated the selection step, or can be 
, performed independently in response to other needs. 



; / / 

Selecting A Storage Image 

to each of the file system image transfer techniuues, the method 300 selects 



a 



storage image 220 to be transferred. 

A, a flow point 370, the file server 1 10 is ready to select a storage image 



220 for transfer 

25 



Atastep 37!, the file server 110 forms a logical sum LS of a se, of storage 
images 220 A. ♦ A2, mus LS - Al + A2. The logical sum LS can a,so include ^ 
p.urality of storage images 220, such as Al + A2 + A3 + A4, mus for example LS 
+ A2 + A3 + A4. 

At a step 372, the fi.e server U0 determines if the transfer is a Ml transfer 
or an incremental transfer. ,f the transfer is incremental, the method 300 continues w„„ 
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At a step 373, the file server lin r 
> 'ogica, sum LS and a base storage i mage 220 R ^ * ^ LD ** 

i-age 220 B comprises a snapshot 210. ' ^ = LS ~ * ^ baSe Stora ^ 

At a flow point 380, the file server 11 n k , 
transfer. 1 1 0 has ^ted a storage i mage 230 for 

10 

Volume Copying 

At a flow point 310, the file server lin- , 
copying operation. ° 15 read y t0 Perform a volume 



15 



At a step 311, the file server 111 i 
described with regard to the flow point 370 th u ^ ** ^ M 

-Pying operation is a fi.1, ^ ^"J^ ^ *" ^ 380. Ift hevolume 
t-sfe, If the voIume ' St0 ^ -ge 220 selected is for a full 

» ^^i. fc .lLS^ lncremental VOIurae the — 

A ' a step 312, the file server 1 inn.,. • . 
<o disk or t „ (ape. ° daem " nes lf ™lume is ,„ be copied 

0 ,fthCVOlUmeiS,Ob "^«°^-^300co„ tinU e sw , te 

At a step 313, the file server 11 n 
30 selected storage image 220 l„ aDrf . * ^ Stream 230 the 

s.rea™ 230 are „ r , red for IT.^tT-T T ^ ^ ^ 

K - Each stora ge block 1 15 is associated 
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with a VBN (virtual block number, for identification. The -hod 300 — with 

the step 315. 

At a step 3 14, the file server 1 10 performs the same tactions as in the step 

. -no are ordered for transfer 

3,3, except that the storage blocks 115 in the rmage stream 230 are 

to tape. 

At a step 315, the file server U0 copies the image stream 230 to the 
destination file system 120 (disk or tape). 

0 lf the image stream 230 is copied to disk, the file server 1.0 preferably places 
the source disk(s), similar to what wouldhappen on retrieval from tape. 

,„ a preferred embodiment, the file server 1 10 copies the image stream 230 

,o acknowledgement or retransmission of data. 

^ destination file system 120 presents the image stream 230 directly to a 
reslore element, which copies the image stream 230 — 
^etdis^astheywereonthesourcedisUs). ^"^^^ -5 

„ from the file server 1 10 to the destination file system 120, the storage 
25 copied from the me server ^ 
in the image stream 230 can be used directly as a consistent file system 

arrive at the destination file system 120. 

The destination file system 120 might have ,„ alter some inter-block 
Le to the VBN of each storage block 115, if some or all of the target 
3 „ pointers, response to W ^ ^ ^ ^ ^ 

storage blocks 115 are recorded in different pnysica 

storage blocks 115. 
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If the image slream 230 i s copied t0 

-* borage b.oc k , I5 h , " ' «~ I -0 preferab ly places 

* "» VBN. When U, e slorage b|ocks J* **« *■» » «« » can be relieved 
» «* file se„er „ 0 , ^ are , *" ™«««d from ^ lnt0 

^ "*« « were on source " a ' U>V * k "' °" * 

The destination file system 120 
«Pe, along with a se, „f block number ^ *• ™* •**» 230 direc t | y „ nl0 
•taft-ta file system 120 can laler _ ° ma " 0n *» each ■«* block „5. The 

» - o„ (0 a d isl Bk JTi^Z S ' 0ra8e ^ ' " f ™ * - 

*» »e fl ,e server , ,0 ,„ ,„ e desll , ation ffle <* 1 .4 * copied 

"" age S,ream »> ean be restored diredv to A " ' ^ Wocks 1 15 » *e 
^nation file system 120 . " he " ,a,er re «eved from rape auhe 



15 



The destination file system 120 . . 
Pointers, responsive to the VBN 0 f each « * ^ *** inter - bIo <* 

-a 8 ehlo Ck s n 5 ,eret ri eved fr I;;:? 6 « al, of the tar get 

the so.ce stora geb l oc C e rr^^ 
Nation in header data that ^ ^ 



At a flow point 320, the file server! 10 h, c , 
operate. W 1 10 has C0 ™P'eted the volume copying 

Volume Mirroring 



A t a flow P0 i nt 330) the fi] 
mirroring operation. 1 10 ,s read y to perform a volume 

Atastep 331, the file server lin„„r 
- bribed w iln regard t „ * flow ^ ^ * *» co Pying ^ 
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0 If the function to be performed is fell mirroring, ,he file server 1 10 performs the 
fu„ volume copying operation to disk as the targe, destination file system 120. 

0 lf the function to be performed ,s incremental mirroring, the file server 1.0 
performs the ft* volume copying operation to tape as the target des.ina„o„ file 

system 120. 

At a step 332, the file server 110 sets a mirroring timer for incremental 
update for the volume mirroring operation. 

A, a step 333, the mirroring timer is hit, and the file server 1 10 begins the 
incremental update for the volume mirroring operation. 

At a step 334, the file server 110 performs an incremental volume copying 
15 operation, as described with regard to the flow point 3.0 through 4c flow point 32. 
The vdume copying operation is performed for an incremental upgrade of the file system 

114. 

The incremental volume copying operation is performed with disk as the 

20 target destination file system 120. 

0 ,f the initial full volume copying operation was performed to disk, me destination 
fil e system .20 increments its copy of the fi.e system 1.4 to mclude the 
incremental storage image 220. 

25 0 ,f the initia, full volume copying operation was performed to tape, the destination 
file system 120 records the incremental storage image 220 and integrates.. ,n.o an 
.ncrelenta, mirror da. structure, as described above, for possmly later 

incrementing its copy of the file system 1 14. 
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At a step 335, the file server 110 c • 
destination file system 120. The method ,nn ^ ^ 23 ° '° the tar * et 

ng timer, and (he method 300 continues. 

When the destination file svst ,_ ,,„ . 
records the storage biocks 1 15 i„ ^ im j the taa ^ «"» 230, i, 

copying, „ dcscrib£d wj|h ^ tQ J* 3 S - m 230 simiiar to the process of v„, ume 

If the method 300 is haIt P H ru 
- -hod 300 completes at the flow point JT " ™ W the 

At a flow point 340, the file server II n k 
mirroring operation. aS com Pleted the volume 

15 Reinte ^ ration of Incremental Mirror 

At a flow point 370 th* r,i~ 
«» <»= st „ ra8e J ge *™ "° " «* <° ™« a a. system 

g and (he mcremental mirror data structure. 



20 A 

system. 3 "'' fi ' e ^ ' 10 ~* - »- storage image 220 



into its 



At a step 372, the file server 1 1 0 read, th, ■ 
into its file system and uses th at data struct ^ mCren,entaI ™™ da * structure 

13 StmCtUre t0 "P date ^ base storage image 220. 
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At a flow point 380, the file server 1 10 i.r,^ , 

^ ^ ~ from the base storage image 2 1 ^ * 
structure. 8 mage 220 ^ th e incremental mirror data 
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15 



Volume Replication 

At a flow point 350, the file server 110 is ready to perform a volume 
replication operation. 

At a step 351, the destination file system 120 initiates the volume 
replication operation. The destination file system 120 sends an indicator of its newest 
master snapshot 210 to the file server 1 10, and requests the file server 110 to perform the 
volume replication operation. 

At a step 352, the file server 1 10 determines if it needs to perform a volume 
replication operation to synchronize with a second file server 110. In this case, the 
second file server 110 takes the role of the destination file system 120, and initiates the 
volume replication operation with regard to the first file server 1 10. 

At a step 353, the file server 110 determines its newest master snapshot 
210, and its master snapshot 210 corresponding to the master snapshot 210 indicated by 
the destination file system 120. 



20 O 



25 



30 



If the file server 110 has at least one master snapshot 210 older than the master 
snapshot 210 indicated by the destination file system 120, it selects the 
corresponding master snapshot 210 as the newest one of those. 

In this case, the method proceeds with the step 354. 

If the file server 1 10 does not have at least one master snapshot 210 older than the 
master snapshot 210 indicated by the destination file system 120 (or if the 
destination file system 120 did not indicate any master snapshot 210), it does not 
select any master snapshot 210 as a corresponding master snapshot. 

In this case, the method proceeds with the step 355. 
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At a step 355, the fil c se „er 1 10 performs a volume coovir,. 
response ,o .he newest master snapshot 2,0 i, has avai,ab,e T 7 
with the flow point 360 " haS aVailabk - ™« method 300 proceeds 



At a flow point 360, the file server lift i, 
rep.ica.ion operation. Tfce destination ffle svsJu I ^ ** ^ 

correspond to the raas ,er snapshot 2,0 * 1 h " SnaPSh °' 

from the fi,e server , ,0 '° "* ^ ^» ^sfer 
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inventorsna.lrihlT" ^ ^ - the 



Alternative Embodiments 
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Although preferred embodiments are disclosed im- 
possible which remain within h, ' ^ Variations are 
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Claims 



1 A file system, having a plurality of storage blocks, and including a 
plurality of bits associated with each one of said plurality of storage blocks, at least one 
of said plurality of bits identifying whether said one storage block was part of sod file 
system at a time earlier than a current consistent version of said file system. 

2 A file system as in claim 1, including a second one of said plurality 
of bits identifying whether said one storage block was part of said file system at a second 
time earlier than a current consistent version of said file system 

3 A file system as in claim 2, including an element disposed for 
selecting storage blocks in response to said one bit and said second one bit associated 
with said selected storage blocks. 



5 

4. 



A file system as in claim 3, including an element disposed for 
copying said selected storage blocks to a destination. 

5 A file system as in claim 4, wherein said destination includes: a 
M tape, a disk, a data structure in a second file system, a set of network messages, or a 

destination distributed over a plurality of file systems. 

6 A file system as in claim 1, including an element disposed for 
selecting storage blocks in response to said one bit associated with said selected storage 

25 blocks. 

7. A file system as in claim 6, including an element disposed for 
copying said selected storage blocks to a destination. 
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8 A file system as in claim 7, wherein said destination includes: a 
tape, a disk, a data structure in a second file system, a set of network messages, or a 
destination distributed over a plurality of file systems. 
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— . ^i"::?::::? of s,ora8e b ^ - fi,e *- 

pl-„,y, said member SIOrage 8 b ^ «°* —d from said 

' «" «'e system „ ,„ a ^ ° bjK ' » - «* w^rein 

snapsho,. filC SySKm re » u «< regard ,„ said 

10. A file system as in claim 9, including 
a mark-on-allocate imaee ofj m „(„. ,. 

from said piurality. member stor "* e w «*s selected 

5 wi,„ regar(l " ^ " daim «" image is defined 

!og,cal sum „pe ra „ 0 „ on said snapsho( and ^ ^ ^ 

12. A file system as in claim 9, including 

fromsaidplu^nZT" ^ " 

ra> * s d member storage blocks having ^ ^ ^ ^ 

deallocate J^ZZ? ~ * - - - 

from said p.ui ^ '"^ 3 " ° f -rage b ,„ ck s selected 

13. A file system as in claim 9, including 

- acive file system, wi m 1 otel d mlT 3 " ^ ^ 
consisten, filesystem; and ' "'^ ~ fr » «*< 

— jjzztr?™ m response ,o said s " apshot - -* -» 

ptaLy. 85 Ca " n8 3 *" ° f membCT Elected *» said 
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14. A file system as in claim 9, including an indicator of which ones of 
said member storage blocks have been copied. 

15 A file system as in claim 9, including a plurality of said snapshots; 
wherein said plurality of said snapshots are associated with an array of bits, said array 
having one set of bits for substantially each storage block in said plurality of storage 
blocks, said set of bits having at least one bit for each said snapshot. 

16 A file system as in claim 9, wherein said file system can manipulate 
said snapshot without having to traverse a hierarchy of file system objects within sa.d 

snapshot. 

17 A file system as in claim 9, wherein said snapshot includes a data 
structure disposed in a forma, allowing for a set management operation to be performed 
relatively efficiently. 

18 A file system as in claim 9, wherein said snapshot includes an array 
of bits, said array having one bit for substantially each storage block in said plurality. 

0 19. A file system as in claim 9, including 

a plurality of said snapshots; and 

a storage image determined in response to said plurality of snapshots; 
said storage image defining a second se, of member storage blocks selected 

from said plurality. 

25 • „io? m io inHudine an indicator of which ones 

20. A data structure as in claim 19, mciuaing <w 

ofsaid storage blocks in said storage image have been copied. 

21 A file system as in claim 19, wherein said storage image is a result 
30 0 f a logical sum or difference performed on said set of member storage blocks for sa,d 
snapshot and a set of member storage blocks for a second said snapshot. 



37 



WO 00/07104 

PCT7US99/I7148 

25. A file system as in claim 24 wh^in * -a 
is a logical sum or difference. mana S eme °< operation 

said snapshot. . Sa ' d "'"""O' a « member storage blocks cf 

™re than abouU/Xf IT " ^ * Where ' n "* ^ ~ « "° 

L A ° f OPCratinS 3 fi ' e S3id ■»** -'"ding steps for 

— -sczx— - r - - ■ • 

first consistent state- 3 Set ° f St0ra § e b,ocks * said 
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forming a second snapshot of a second consistent state of said file system, 
said second snapshot induding an indication of a set of storage blocks in said second 

consistent state; and 

performing an operation on said firs, and second snapshots to form 
storage image including an indication of a, least some storage blocks in said fiie system. 



30. A 
sum or difference. 



method as in claim 29, wherein said operation includes a logical 



31. A method as in claim 29, wherein 

said operation includes a logical sum or difference; and 

said purpose includes making a copy including or excluding a selected 

range of snapshots. 

32. A method as in claim 29, wherein 

said operation includes a logical sum or difference; and 

said purpose includes copying said storage image to a destination. 

33 A method as in claim 32, wherein said destination includes a tape, a 
disk, a data structure in a second file system, a set of network messages, or a destmauon 
distributed over a plurality of file systems. 

34. A method to be performed in a file system, said file system having a 
plurality of storage blocks, said method including steps for 

defining a storage image of a set of member storage blocks selected from 
said plurality, said storage image being formed in response to a set of member storage 
blocksformin g aconsistentfaesystemothermananactivefilesystem;and 

forming an image stream of a sequence of member storage blocks se.ected 
from said storage image. 

35. A method as in claim 34, including steps for associating a block 

location with substantially each one of said sequence. 
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36. A method as in claim 34 h ■ 
-on S1 ^„ g ame SysI e minrespomelosa . d . mag JJ™ ™< <*«*" includes 

37 - A method as in claim id u * 
> Permed in reS p„„ se t0 , sekc[ ^ *• Whm ' n said «I» for forming are 

38 * A method as in claim *a , u - 

1 0 dge Wocks fo r a file system operation. 

39. A method as in claim ia , u 

- optimi , ng ^ sequence of - - ing incIude 

m a RAID file system. ° r 3 file s y stem operation 

5 steps for " ^ " ta * -id steps for f orming inc , ude 

paraHe,; and *«■* capable of being read in 

- ~rzzzzzzz? r - - - — 

storage elements. "> parallel from said plurality 0 f 

complete f„e sys,e ra A ""^ " " **" * Where ''" S,0ra 8 e *-* -P-sents a 
of changes to ale ^ " " ^ ^ ^ *"* ^ — • « 
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periodic intervals. 
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44. A method as in claim 34, including repeating said selecting step m 
response to an operator command. 

45. A method as in claim 34, including repeating said selecting step in 
5 response to a remote device. 

46 An incremental mirror copy of a file system, said incremental copy 
including a base set of storage blocks stored in a first storage medium, and an 
incremental set of storage blocks stored in a second storage medium. 

10 

47. An incremental mirror copy as in claim 46, wherein 

said first storage medium is substantially slower than said second storage 

medium; and 

said incremental set of storage blocks is more recent than a tune needed to 
15 recover said base set of storage blocks. 

48. An incremental mirror copy as in claim 46, wherein said incremental 
set of storage blocks is responsive to a plurality of updates of said file system. 
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49 An incremental mirror copy as in claim 46, wherein said incremental 
se, of storage blocks is responsive to a continuous sequence of updates of said file 
system, wherein said incremental mirror copy includes a substantially up to date se, of 
storage blocks in said file system. 

50 An incremental mirror copy as in claim 46, wherein said incremental 
set of storage blocks is responsive to an indication of a set of storage blocks deallocated 
from said file system. 

51 Apparatus including 

a file system including a plurality of snapshots thereof, each representing 
associated consistent state at an associated selected time; and 
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an 
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said associated consistent state. * °" e S,ora S e b, °* 

« leas, two of said snapshot, ^ » a " ~ on 

- said iiL: :::;i;:;;r of a fi,e system having a ^ ° f — 

system; and ?Y " fet COns,stent ve ™on of said file 

■o a second consistent JLT^T ^ ** ~ — - 

- g s,o re di;::::;::~ :; a r s,orase - — - 

rage medium of substantially different type- 

^-jsr 1 copy of said B,e system - te ~ - - 

storage medium. ? * ' S *«« tnan said second 

55- An incremental mirror copy as in d»i m « ■ 

of member storase blocks is rcsponsive to a P1 :: i; j:;x:;rrL^r nd set 
-~ . i citsr .r: ,n c,a,m 53 ^ - — - 

P to a continuous sequence of updates of said file 
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system, wherein said second set of member storage blocks includes a substantially up to 
date set of storage blocks in said file system. 

57. An incremental mirror copy as in claim 53, wherein said second set 
of member storage blocks is responsive to an indication of a set of storage blocks 
deallocated from said file system. 

58. In a file system having a plurality of storage blocks, a data structure 

including 

a first snapshot of a set of member storage blocks selected from said 
plurality, said member storage blocks forming a consistent file system other than an 
active file system; 

said first snapshot being represented as an object in said file system and 
having a set of storage blocks for recording said first snapshot; 

whereby copying said member storage blocks in said first snapshot has the 
property of preserving at least one snapshot recorded in said file system at a time of said 
first snapshot. 

59. A data structure as in claim 58, including 

a second snapshot of a set of member storage blocks selected from said 
plurality, said member storage blocks forming a consistent file system other than an 
active file system; 

said second snapshot being represented as an object in said file system and 
having a set of storage blocks for recording said second snapshot; 

whereby copying said member storage blocks in said second snapshot has 
the property of preserving at least one snapshot recorded in said file system at a time of 

said second snapshot, 

60. A data structure as in claim 58, including 

an image stream including a set of storage blocks including both said first 
snapshot and said second snapshot; 
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whereby copying said member storage blocks in *»ia • 

™ o f „ g both m ta snapshot jj2:zzr s,ream has thc 

5 including InaflleSyS,emhaVi "^'' 1 -"'-f^eb l oc ks , ada , astnlcture 

said m h 3 SnaPSh °' ^ 3 " ° f raember ^ bi ° CkS *» -id p, urality 

storage formmg . consjslem fife sysam o[her to m ^ 

including M ' 3 fife SyS ' em haVi " 8 3 PlUraUty * "«* • -«u re 

p,u ra .itv; 3 ^ ^ ° f 3 " ° f *» said 

said s. orag = image being fom , ed jn rKponse 
blocks f or m mg . consistent fi , e sysKm oiher(han ^ act . ve ^ 

63. A dala structure as in claim 62, including 
consistent ^ ^ 3 " " — — ' "* ' 

a sequence of increment storage images, e a ch having a predecessor „ 
leas, one of said predecessors being said firs, storage image- 

completed ' ^ " * "*» >— »— - - one 



of h , Ada,3StmC,Ure 35 i" claim 62, induding an indicator of which ones 

of sa,d storage blocks in said storage image have been copied. 
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65 A data structure as in claim 62, wherein said storage image indicates 
a logical difference of two sets of member storage blocks, at least one of said sets 
forming a consistent file system. 

66 A data structure as in claim 62, wherein said storage image indicates 
a logical sum of two sets of member storage blocks each collectively forming a 
consistent file system. 

67. A data structure as in claim 62, wherein said storage image indicates 
10 a set of member storage blocks forming a consistent file system. 

68 In a file system having a plurality of storage blocks, a data structure 
including a shadow snapshot of a set of member storage blocks selected from said 
plurality, said member storage blocks having formed a consistent file system other than 
15 an active file system, with a set of selected member storage blocks removed from saui 
consistent file system. 

69 A data structure as in claim 68, wherein said shadow snapshot is 
deposed in a format allowing for a sc. management operation to be performed relatively 

20 efficiently. 

70 A data structure as in claim 68, wherein said shadow snapshot uses, 
in addition to said member storage blocks, no more than about 1/100* of an amount of 
storage required by said storage blocks in said plurality. 
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71 A data structure as in claim 68, wherein said shadow snapshot uses, 
to addition to said member storage blocks, no more than about one byte per storage block 
in said plurality. 

72 A data structure as in claim 68, wherein said shadow snapshot is 
disposed as a single object in said file system, whereby said file system can manipulate 
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said snapshot without having to traverse a hierarchv of fil 

snapshot. Chy ° f fi,e s y stem objects within said 

73. A data structure as in claim u ■ 

74. A data structure as in claim 7" u 
includes a file system operation. ' " ^ processin 8 Ration 

75. A data structure as in claim 73 h 

76. A data structure as in claim a» u • 

disposed fa a f„ mal aIlowing for a sa " Where '" ^adow s Mpshot is 

*» o, iess, where . is a „„1 0 r g :~ T- '° * <*> 
— of said s,o rag e biocks , JST" ' ^ Wi ' hOU ' ^ ^ 

77. A data structure as in H a ; m 
operation is a iogica, sum „ r djfference ^ ?6 ' " h « ta Said « — ge m e„t 

- — . storage i j h :r:;:r b,ocks sekcted &om saM 

formed a consist fife system . 8 ^ ^ '° a SM J> sh °< *■ origUy 



79. A data structure as in r\*i m no 
storage image is disposed as a <- . u . ?8 ' Wherein said m ark-on-al locate 

- -pie JZZ^^ me ^ W ^^ 
objec* w ithi „ said snapsh l " hOU, haV '" g » *"« * of fi ,e s ysl e m 
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80. A data structure as in claim 78, wherein said mark-on-allocate image 
is disposed in a format allowing for a set management operation to be performed 
relatively efficiently. 

81. A data structure as in claim 78, wherein said mark-on-allocate 
storage image uses no more than about 1/100* of an amount of storage required by said 
storage blocks in said plurality. 

82. A data structure as in claim 78, wherein said mark-on-allocate image 
uses no more than about four bytes per storage block in said plurality. 

83. A data structure as in claim 78, said member storage blocks having 
been selected responsive to completion of a processing operation. 

84. A data structure as in claim 83, wherein said processing operation 
includes a file system operation. 

85. A data structure as in claim 83, wherein said processing operation 
includes reuse of said selected member storage blocks by said file system. 

86. A data structure as in claim 78, wherein said mark-on-allocate image 
is disposed in a format allowing for a set management operation to be performed in O(n) 
time or less, where n is a number of storage blocks in said plurality, without readmg any 
contents of said storage blocks in said plurality. 

87. A data structure as in claim 86, wherein said set management 
operation is a logical sum or difference. 

88 In a file system having a plurality of storage blocks, a data structure 
30 including a mark-on-deallocate image of a set of member storage blocks selected from 
said plurality, said member storage blocks having been removed from a snapshot that 
originally formed a consistent file system. 
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89. A data structure as in claim us u ■ 
-age image is d i sposed as . singfc J «™ 1 m " k -°*-™°™ 

maniPUIate aid snapshot wL/;: : t ;:r; ? saidfi,esystem 

objects within said snapshot. * ° f Rk ^ 

5 

> * :i::ri;j;;r 88 ' wherei - s * d 

relatively efficiency. manageraem 0perati °" 10 * Permed 

— . -,1 „:;: rir,;;;: 88, wherein saw mark -°— 

s-orage Weeks in said piuraiity. " am ° Um ^ <» >* 

™a 8 e U se S „:ir::r;:;" b ;r said mark — 

oyies per storage block in said plurality. 

- - - jl n^vr^ wherein sau mark -°- d — 

in O(n) time or less wh • m ^ge m ent operation to be performed 
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AMENDED CLAIMS _ nol0 o y 
remaining claims unchanged (8 pages)J 

„eeded to recover said base set of storage blocks. 

in Haim 46. wherein said 
48 An incremental mirror copy as m claim 4 . 

system. 

oc in Haim 46. wherein said 
49 An incremental mirror copy as in claim 

' , ce blocks is responsive to a continuous sequence of updates 
10 incremental set of storage blocks is respo substan tially 
of said file system, wherein said incremental mtrror copy includes 
up to date set of storage blocks in said filesystem. 

50 An incremental mirror copy as in claim 46, wherein said 
blocks deallocated from said file system. 

51 Apparatus including 

to slm tnc.uding a plu* of snapshots thereo -* 

each satd snapshot — an -cattor » ^ 
in M associated consent state, at ieast « storage ^ - J ^ 
row and a column bit plane intersection, sa,d ,nd,a„on be.ng 
one storage block in said associated consistent state. 

5 , Apparatus as in claim 51, including a storage image defining at 
, £ast ^eblcl in satdftle .stem, sa,d storage image response to an 
operation on at least two of said snapshots. 

f . file svstem havina a plurality of 
53. An incremental minor of a file system 

storage blocks, said incremental mirror including 
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a first set of first member storage blocks select r 
first member storage blocks formin. a ^ ' ^ "* 

file system; and " ™ C ° nS ' Stent versio " of said 



consistent version and to , , 8 TeSpmsive t0 said fir * 

ersion and to a second consistent version nf c -a 

second set including a sef of u * SyStem ' said 

aing 3 set °f changes between sairf f?™t j 
version; ' d firSt and sec ond consistent 

ora & e medium of substantially different type- 
whereby a complete copy 0 f said ffle ' 
said first set and said second set constructed from 
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An incremental mirror as in claim « , k • 
storage medium has much greater sto™ • ^ Said fim 

55- An incremental mirror copy as in claim « k • 
second set of member storage blocks is resnn ^ 
20 file system * 3 pluraiit >' of updates of said 



5 



«• An incremental mirror copy as in claim « i, • 
s«o„d « „f member Py cl <™ «. where,„ said 

of said file system * ~* » a sequence of 

c system, wherein sa d second 

— - ;r:rr copy 35 m cwm * ** - 

-se^dea^edfi:::;;;::- 10 - — — - 
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» p,ane in.ersec.ion, said member storage blocks fonmn. 
other than an active file system; file system 

said firs, snapshot being represent as an obj 

has the property of preservin D 
time of said first snapshot. 

59 . A datas.ruc. m easinc,aim58,inc>nding 
a SK ond snapsho. of a se, of member * 
saia p,ura,i„sa,d member storage b,ocks formmg a — files, 

taa " C,,Ve rd S ^ snapshot being represent as an object in s.,d file 

whereby copying said member storage blocks 

reserving at least one snapshot recorded m sa,d file syste 
has the property of preserving at 

time of said second snapshot. 

A data structure as in claim 58, including 
Team — ase, of storage blocks including both sa, 
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60. 

an image 



i outv nf storage blocks, a data 
61 . m a file system having a plurality of stora. 

30 structure including 
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a snapshot of a set of m „ u 
other than an active file system- 8 " C ° nSiStent fi,e S * stem 

a storage imase of a set nf ™~ u 

63- A data structure as i n claim 62. including 
a first storage i mage indicating a sef nf u 
> forming a consistent file system- and " St ° ra8e bl ° cks 

wherein a logical sum of said set of «t« • ' 
one complete snapshot. ^ ' mages incIudes « 'east 

6 4- A data structure as in claim • . ^ 

«. of said s,„ rage blocks in ^ *™ «■ - W-or of . hie 

t0rage ,ma ge have been copied. 

65 - A data structure as in claim u - 

«ca te a iogica, difference of two _ of „ " *** 

-« « *»*g a cc, sistMI file ^ * - - one of 
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a * in claim wherein said storage image 
^ A data structure as in claim o-, 

forming a consistent file system. 

« i„ claim 62 wherein said storage image 

67 A data structure as in claim oi, 

ki„,-ls forming a consistent file system, 
indicates a set of member storage blocks forming 

68 ,„ a file system having a plurahty of storage biocks, a data 

\ shot of a set of member storage blocks selected 

structure including a shadow snapshot of . ^ ^ 

a column b,t plane intersection, said member storage 
eonsisten. file system other than an active f,!e system, w„h 
Ilrstorageblocksremoved from said consistent fi.e system. 

69 A data structure as in claim 6S, wherein said shadow snapshot 

relatively efficiently. 

70 A data stmcture as ,n Cairn 6S, wherein said shadow s„ap« 
20 uses , a ddit,on,osaidme m berst„rage W oc k s,n„more,a„abou. 1 nOO 
IL,ofs,oragere q uiredb y saids,orageb,oc k sinsaid P ,ura,.,. 

• i ™ f.9. wherein said shadow snapshot 
7 ! A data structure as m claim 68, wherein 

storage block in said plurality. 

• i - m <;» wherein said shadow snapshot 
72 A data structure as in claim 68, wherein 

- 'A file svstem whereby said file system can 
1S dis posed as a single object m said file ys.m ^ ^ ^ 

30 ma nipulate said snapshot without haymg to traverse 
objects within said snapshot. 
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73 - A data structure as in claim fix „,i. ■ . 

74 - A data structure as in claim 71 u ■ 
opera,™ mdudes . file syslem ^ «• *-„ said process,,* 

75 - A data structure as in claim 71 ,. . 

operation inciudes reuse of said selected 7 Pr0CeSSi " 8 
■0 system. W ' ed membe ' *"* "locks by said file 

T>- Adatast ™amea s i„c|ai m68 , wherei 

op^on is a ,ta,tl:;:~ data * - - 

— a" a r* of s,orage b, °*- * *• 

^ected front said p,„ r a,i,y, sa , d ~ f " ' " " «** 

— , h a,or^,y fo:; d i::::~ ha ™ s - - • 

79 - A data structure as in claim 7« , u ■ 
"orage intage is disposed as a si„ s ,e ob ^ 
system can ma„i P u, ale said ' 7 ' " ""' * said ffle 

— wL: ;:::„x 1 havms ,o — a — * * * 
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SO A data structure as in Cain, 78. wherein said mark-on-allocate 
image is disposed in a format allowing for a se, manage* operation ,o be 
performed relatively efficiently. 

• „i<,;m 78 wherein said mark-on-allocate 
81 A data structure as m claim 78, wtierem m 

said storage blocks in said plurality. 

82 A data structure as in clatm 78, wherein said mark-on-allocate 
, 0 imag e uses no more than about four bytes per storage block ,n said pluraltty. 

83 A data structure as ,n claim 78. said member storage blocks 
ha vi„g been selected responsive ,0 completion of a processing operauon. 

M . A data structure as in claim 83, wherein said processing 
operation includes a file system operation. 

85 A data structure as in claim 83, wherein said processing 

20 system. 

• ,u\ m 7R wherein said mark-on-allocate 
g6 A data structure as in claim 78, wnerem 

imag e is disposed in a fonna, allowing for a 
performed in O(n) time or less, where n ,s a numb, o *™» 
25 plurality, withoutreading any contents of said storage blocks ,n sard plural,*. 

„. A data structure as in claim 86. wherein said set management 
operation is a logical sum or difference. 

88 ,„ a ffle svstem having a plurality of storage blocks, a data 
secure rncWing a mark-on-deallocate image of a se, of member storage blocks 
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^C™LTdT7 ffi in claim 88 ' wherein mw — 

» •* * systen, ma ! , " 3 Si " Sle ° bjeCt Sai " fi ' £ 

"> to be performed relatively efficiently. management operation 

91 • A data structure as in claim a* u ■ 

deallocate storage ima fie uses no u ' ^ Said mark " on - 

* ge uses no more than about 1/1 00 th «r „„ 

^ by said storage b|ocks ,„ ^ ^ 
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plurality 
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93 - A data structure as in claim stst u . 

•o be perform i„ 0,„, , lnK or Iess , ^ „ ^ £ f ° P£ra ' i0n 
Plurality, without ra di„ g any contenls " ° f ^ bl °* s •* 

6 ny contents of sa,d storage blocks in said plurality. 

— ' s a zjztz: : M c,aim * *■* " d - — 
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tndependen, Cairn 51. 58, 61 , 62, and 68 are amended to include a phrase 
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